This invention relates to an image processing device, an image processing method, a medium, a contour extraction device and contour extraction method, and for example, to an image processing device, an image processing method, a medium, a contour extraction device and contour extraction method which enable easy processing such as three-dimensional editing of a two-dimensionally displayed image of a three-dimensional object.
Conventionally, various methods have been proposed for carrying out various processing and extracting information necessary for carrying out various processing from a two-dimensional image. Such methods are described, for example, in the following documents: James D. Foley, Andries van Dam, Steven K. Feiner, John F. Hughes, xe2x80x9cComputer Graphics, principles and practice,xe2x80x9d ADDISON-WESLEY PUBLISHING COMPANY, 1996 (hereinafter referred to as Document 1); Paul E. Debevec, Camillo J. Taylor, Jitendra Malik, xe2x80x9cModeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach,xe2x80x9d proceedings of SIGGRAPH 96, pp. 11-20 (hereinafter referred to as Document 2); Oliver Faugeras, xe2x80x9cThree-dimensional computer version,xe2x80x9d The MIT press (hereinafter referred to as Document 3); Kenneth P. Fishkin, Brian A. Barsky, xe2x80x9cA Family of New Algorithms for Soft Filling,xe2x80x9d proceedings of SIGGRAPH 84, pp.235-244 (hereinafter referred to as Document 4); Pat Hanrahan and Paul Haeberli, xe2x80x9cDirect WYSIWYG Painting and Texturing on 3D Shapes,xe2x80x9d proceedings of SIGGRAPH 90, pp.215-233 (hereinafter referred to as Document 5); Youichi Horry, Ken-ichi Anjyo, Kiyoshi Arai, xe2x80x9cTour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image,xe2x80x9d proceedings of SIGGRAPH 97, pp.255-232 (hereinafter referred to as Document 6); and Michael Gleicher, xe2x80x9cImage Snapping,xe2x80x9d proceedings of SIGGRAPH 95, pp. 183-190 (hereinafter referred to as Document 7).
In Document 1, a image processing technique called two-dimensional paint is described in which the work similar to painting a picture on paper by a designer or the like using a paintbrush or air brush (a technique for painting a picture by scattering a powder paint) is carried out by a computer.
However, in the conventional 2D paint technique, even if a displayed image is of a three-dimensional object, the image itself is handled on a two-dimensional plane. Therefore, if any character is drawn or any pattern is added not in consideration of the direction in the three-dimensional space of the three-dimensional object displayed as a two-dimensional image, the image will be unnatural.
Specifically, in the case where a three-dimensional object imitating a house is displayed as a two-dimensional image as shown in FIG. 1A, if characters are drawn on the wall parts of the house without considering their directions, the characters do not look as if they were drawn on the walls, as shown in FIG. 1B. Similarly, if a rectangle is drawn on the wall part of the house so as to add a rectangular-parallelepiped room without considering the direction of the wall, the image will be unnatural, as shown in FIG. 1C. Moreover, in the case where a column is displayed as a two-dimensional image as shown in FIG. 2A, if characters are drawn on its lateral side ignoring the direction of the lateral side, the characters do not look as if they were drawn on the lateral side of the column, as shown in FIG. 2B.
Therefore, in order to prevent an image from being unnatural in carrying out 2D paint, it is necessary to draw characters or pattern while transforming the characters or pattern in accordance with the direction of a three-dimensional object displayed as a two-dimensional image. However, the operation for such drawing requires some skills.
Thus, there is proposed a method for inputting the inclination or the like of the three-dimensional object displayed as a two-dimensional image by a user using a ten-key pad or a GUI (graphic user interface) and transforming characters or pattern to be newly drawn on the basis of the input by a computer. In this method, however, the user needs to adjust the inclination of the three-dimensional object to be inputted to the computer while watching the drawing result so as not to make the image unnatural, and this adjustment also requires certain skills.
As described above, if the user draws any character or pattern without considering the direction of the three-dimensional object displayed as a two-dimensional image, a drawing result that looks natural, that is, a drawing result that looks as if the object were shot with the character or pattern drawn thereon, cannot be obtained for the following reason. That is, there is lacked information concerning where in the three-dimensional space the three-dimensional object displayed as a two-dimensional image is positioned, and from which position the object was shot to obtain the two-dimensional image (if the two-dimensional image is obtained by shooting a picture, the position where the scenery shown in the picture was observed).
Thus, there is proposed a method for finding the position in the three-dimensional space of the three-dimensional object displayed in the two-dimensional image and the shooting position of the object, from the two-dimensional image by using an image recognition (computer vision) technique.
Specifically, Document 2 discloses a method for associating a three-dimensional object such as a rectangular-parallelepiped with a building shot into a plurality of photographs by using a GUI and then finding the size and the shooting position of the building. Document 3 discloses various other methods for finding the position of the three-dimensional object and the shooting position thereof
Such methods for calculating the position of the three-dimensional object and the shooting position thereof using the image recognition technique utilizes the principle of triangulation. Therefore, a plurality of images obtained by shooting the same three-dimensional object from a plurality of shooting positions are required. However, it is not necessarily possible to prepare such plural images in carrying out 2D paint, and if the two-dimensional image is obtained by shooting a picture, a plurality of images as described above generally do not exist.
Even in the case where a plurality of images obtained by shooting the same three-dimensional object from a plurality of shooting positions can be prepared, the corresponding positions of the same three-dimensional object displayed in the plurality of images (e.g., the apex of the roof of the building as the three-dimensional object) must be designated in order to calculate the position of the three-dimensional object and the shooting position thereof from the plurality of images by a computer. Such operation for designation with respect to each of the plurality of images is troublesome and time-consuming. Moreover, in carrying out natural drawing in a three-dimensional manner on the basis of the position in the three-dimensional space of the three-dimensional object displayed as a two-dimensional image and the shooting position thereof, three-dimensional data such as three-dimensional coordinate of the three-dimensional object as viewed from the shooting position must be processed despite that the two-dimensional image is handled. Therefore, an extremely large quantity of calculation is required.
Document 4 discloses a color processing method in carrying out 2D paint.
Specifically, in 2D paint, for example, the user selects a color to be used for drawing by using a GUI and carries out drawing with the selected color. However, the color and brightness of a three-dimensional object displayed as a two-dimensional image change in accordance with the relation between the direction of the side and the position of a light source. Therefore, for example, if one side of the three-dimensional object is entirely painted with the same color (the same RGB (red, green, blue) value), the drawing result will be unnatural. To obtain a natural drawing result, it is necessary to gradually change the color to be applied in consideration of the relation between the direction of the side to be painted and the position of the light source. Particularly, in the case where the side to be painted is a curved surface, the pixel value must be continuously changed in order to obtain a natural result and therefore the color to be applied must be selected for every several pixels. Therefore, an extremely large quantity of work is required.
Thus, in Document 4, a method called a tint fill method is described, which enables change of one of two colors if the color of a pixel is obtained by mixing the two colors and the two colors are known. However, while the tint fill method can effectively cope with a portion where the foreground and the background exist in a mixed manner, it is difficult with the same method to handle the shadow of an object or reflection of illumination by the object.
Document 5 discloses a method for transforming and then pasting characters inputted by operating a tablet or a mouse, or a handwritten pattern, to follow a side (surface) of a three-dimensional object displayed as a two-dimensional image.
The method disclosed by Document 5 uses three-dimensional information (e.g., position of the three-dimensional object in the three-dimensional space, shooting position, size, inclination, etc.) concerning the three-dimensional object displayed as a two-dimensional image, and therefore it is necessary to accurately recognize the three-dimensional information by using the image recognition technique as described above. However, in the case where the three-dimensional information is recognized by using the image recognition technique, a plurality of images are needed and the quantity of calculation is increased as described above.
The method of Document 5 can also use a three-dimensional model of the three-dimensional object, produced inside a computer. However, in this case, too, three-dimensional data must be handled and an extremely large quantity of calculation is required therefor.
As a method for producing a three-dimensional model, there is proposed a method in which the schematic shape of a three-dimensional object displayed in a two-dimensional image is extracted from the two-dimensional image by using the image recognition technique and in which a shot image is pasted onto the surface of the shape (texture mapping). A technique using this method to produce a two-dimensional image viewed from a free visual point (shot from a free shooting point) is disclosed in the above-described Document 2. In this technique, however, since the schematic shape of the three-dimensional object displayed in a two-dimensional image is extracted from the two-dimensional image, a plurality of images are needed similarly to the foregoing case.
Document 6 discloses a method for producing from one two-dimensional image an image viewed from a visual point except for the shooting position. In this method, however, only an image viewed from a visual point different from the shooting position is produced, and it is assumed that the object displayed as a two-dimensional image and objects on its periphery are of an extremely simple plate-like shape. Therefore, it is not a big problem if an image viewed from a different visual point is to be proximately produced, but if any character or pattern is to be added to the two-dimensional image, it is difficult to obtain a natural drawing result. Specifically, for example, in the case where a column is displayed as a two-dimensional image, it is difficult to obtain a natural image in which any character or pattern is added to the lateral side of the column.
In carrying out processing on a two-dimensional image, the user is often requested to designate a predetermined characteristic point such as an apex of a three-dimensional object in the two-dimensional image. This designation is carried out by operating a mouse or a pen, but it is difficult to accurately designate the apex of the three-dimensional object in the two-dimensional image. Thus, Document 7 discloses a method in which an edge (portion where the brightness and color change largely) in a two-dimensional image is extracted and in which if the cursor is positioned near the edge, the cursor is shifted onto the edge. In the method of Document 7, however, since all the edges in the two-dimensional image are detected, the cursor might sensitively react to the noise and texture of the two-dimensional image and be shifted to an unwanted position.
As described above, though a number of image processing techniques have conventionally been proposed, there has been proposed no method particularly suitable to the case where editing or processing such as three-dimensional addition, deletion, shift and transformation of a character or pattern, change of a color or pattern, and change of a light source is to be performed on a two-dimensional image.
That is, in the conventional methods, certain operational skills and time are required or a plurality of images obtained by shooting the same object from different directions are necessary to perform three-dimensional editing or processing on a two-dimensional image. Also, increase in the quantity of calculation and unwanted shift of the cursor might occur. Moreover, if a three-dimensional object in a two-dimensional image is deleted, the background must be reproduced in the area after the deletion. However, with the methods described in the above-mentioned Documents 1 to 7, such reproduction of the background is difficult.
In view of the foregoing status of the art, it is an object of the present invention to enable easy implementation of three-dimensional editing or the like with respect to a two-dimensional image.
An image processing device according to the present invention includes: first operation means for being operated when designating first shape information concerning the shape of a side constituting a first three-dimensional object displayed in a first two-dimensional image and a first characteristic point, which is a point on the first two-dimensional image constituting the side; first calculation means for calculating a first conversion formula for converting the side constituting the first three-dimensional object in the first two-dimensional image to a spread image, which is an image spread on a two-dimensional plane, on the basis of the first shape information and the first characteristic point; first conversion means for converting the side constituting the first three-dimensional object in the first two-dimensional image to the spread image on the basis of the first conversion formula; storage means for storing the spread image; second operation means for being operated when designating second shape information concerning the shape of a side constituting a second three-dimensional object and a second characteristic point, which is a point on a second two-dimensional image constituting the side; second calculation means for calculating a second conversion formula for converting the image stored in the storage means to a projection image, which is an image obtained by projecting the side constituting the second three-dimensional object, on the basis of the second shape information and the second characteristic point; second conversion means for converting the image stored in the storage means to the projection image on the basis of the second conversion formula; and paste means for pasting the projection image to a portion specified by the second characteristic point in the second two-dimensional image.
An image processing method according to the present invention includes: a first calculation step of calculating a first conversion formula for converting a side constituting a first three-dimensional object in a first two-dimensional image to a spread image, which is an image spread on a two-dimensional plane, on the basis of first shape information and a first characteristic point; a first conversion step of converting the side constituting the first three-dimensional object in the first two-dimensional image to the spread image on the basis of the first conversion formula; a storage step of storing the spread image into storage means; a second calculation step of calculating a second conversion formula for converting the image stored in the storage means to a projection image, which is an image obtained by projecting a side constituting a second three-dimensional object, on the basis of second shape information and a second characteristic point; a second conversion step of converting the image stored in the storage means to the projection image on the basis of the second conversion formula; and a paste step of pasting the projection image to a portion specified by the second characteristic point in a second two-dimensional image.
A medium according to the present invention causes a computer to execute a computer program, the computer program including: a first calculation step of calculating a first conversion formula for converting a side constituting a first three-dimensional object in a first two-dimensional image to a spread image, which is an image spread on a two-dimensional plane, on the basis of first shape information and a first characteristic point; a first conversion step of converting the side constituting the first three-dimensional object in the first two-dimensional image to the spread image on the basis of the first conversion formula; a storage step of storing the spread image into storage means; a second calculation step of calculating a second conversion formula for converting the image stored in the storage means to a projection image, which is an image obtained by projecting a side constituting a second three-dimensional object, on the basis of second shape information and a second characteristic point; a second conversion step of converting the image stored in the storage means to the projection image on the basis of the second conversion formula; and a paste step of pasting the projection image to a portion specified by the second characteristic point in a second two-dimensional image.
An image processing device according to the present invention includes: operation means for being operated when designating shape information concerning the shape of a side constituting a three-dimensional object and a characteristic point, which is a point on a two-dimensional image constituting the side; calculation means for calculating a conversion formula for converting the side constituting the three-dimensional object in the two-dimensional image to a spread image on the basis of the shape information and the characteristic point; and conversion means for converting the side constituting the three-dimensional object in the two-dimensional image to the spread image on the basis of the conversion formula.
An image processing method according to the present invention includes: a calculation step of calculating a conversion formula for converting a side constituting a three-dimensional object in a two-dimensional image to a spread image on the basis of shape information and a characteristic point; and a conversion step of converting the side constituting the three-dimensional object in the two-dimensional image to the spread image on the basis of the conversion formula.
A medium according to the present invention causes a computer to execute a computer program, the computer program including: a calculation step of calculating a conversion formula for converting a side constituting a three-dimensional object in a two-dimensional image to a spread image on the basis of shape information and a characteristic point; and a conversion step of converting the side constituting the three-dimensional object in the two-dimensional image to the spread image on the basis of the conversion formula.
Another image processing device according to the present invention includes: operation means for being operated when designating shape information concerning the shape of a side constituting a three-dimensional object and a characteristic point, which is a point on a two-dimensional image constituting the side; calculation means for calculating a conversion formula for converting a predetermined image on a two-dimensional plane to a projection image on the basis of the shape information and the characteristic point; conversion means for converting the predetermined image to the projection image on the basis of the conversion formula; and paste means for pasting the projection image obtained from the predetermined image to a portion specified by the characteristic point in the two-dimensional image.
Another image processing method according to the present invention includes: a calculation step of calculating a conversion formula for converting a predetermined image on a two-dimensional plane to a projection image on the basis of shape information and a characteristic point; a conversion step of converting the predetermined image to the projection image on the basis of the conversion formula; and a paste step of pasting the projection image obtained from the predetermined image to a portion specified by the characteristic point in a two-dimensional image.
Another medium according to the present invention causes a computer to execute a computer program, the computer program including: a calculation step of calculating a conversion formula for converting a predetermined image on a two-dimensional plane to a projection image on the basis of shape information and a characteristic point; a conversion step of converting the predetermined image to the projection image on the basis of the conversion formula; and a paste step of pasting the projection image obtained from the predetermined image to a portion specified by the characteristic point in a two-dimensional image.
A contour extraction device according to the present invention includes: parameter calculation means for finding a parameter, of differential values of scalar quantity at a point on a formula f=0 with a function xe2x80x9cfxe2x80x9d represented by using one or more parameters for defining the contour of an object in a predetermined space, the parameter maximizing a normal component, which is a component in the direction of a normal vector at that point; and contour extraction means for extracting the point on the formula f=0 with the function xe2x80x9cfxe2x80x9d represented by using that parameter, as the contour of the object.
A contour extraction method according to the present invention includes: finding a parameter, of differential values of scalar quantity at a point on a formula f=0 with a function xe2x80x9cfxe2x80x9d represented by using one or more parameters for defining the contour of an object in a predetermined space, the parameter maximizing a normal component, which is a component in the direction of a normal vector at that point; and extracting the point on the formula f=0 with the function xe2x80x9cfxe2x80x9d represented by using that parameter, as the contour of the object.